Accuracy of Allplex SARS-CoV-2 assay amplification curve analysis for the detection of SARS-CoV-2 variant Alpha

Aim: To evaluate the accuracy of two PCR-based techniques for detecting SARS-CoV-2 variant Alpha (B.1.1.7). Materials & methods: A multicenter prospective cohort with 1137 positive specimens from Slovenia was studied. A mutation-based assay (rTEST-COVID-19 qPCR B.1.1.7 assay) and amplification curve pattern analysis of the Allplex SARS-CoV-2 assay were compared with whole-genome sequencing. Results: SARS-CoV-2 variant Alpha was detected in 155 samples (13.6%). Sensitivity and specificity were 98.1 and 98.0%, respectively, for the rTEST-COVID-19 qPCR B.1.1.7 assay and 97.4 and 97.5%, respectively, for amplification curve pattern analysis. Conclusion: The good analytical performance of both methods was confirmed for the preliminary identification of SARS-CoV-2 variant Alpha. This cost-effective principle for screening SARS-CoV-2 populations is also applicable to other emerging variants and may help to conserve some whole-genome sequencing resources.

the amplification curve, which is an assay-specific characteristic [12]. These changes can be used as proxy indicators of the presence of the variant concurrent with virus detection, or they can be used retrospectively to analyze the spread of the variant independent of whole-genome sequencing (WGS) [7,13,14].
In Slovenia, systematic non targeted and targeted genomic surveillance of SARS-CoV-2 was introduced in January 2021 by combining WGS of 10% of all PCR-positive samples and screening of another 10% of PCRpositive samples by specific real-time RT-PCR to detect variant Alpha, respectively. With the increasing number of PCR-positive samples during the third wave of the epidemic and the increasing prevalence of variant Alpha, it became necessary to modify the surveillance protocol. The primary aim of the study was to verify whether the specific real-time RT-PCR used in the targeted approach can be used as a confirmatory test for variant Alpha, as claimed by the manufacturer. At the same time, we evaluated the accuracy of amplification curve pattern analysis of the Allplex SARS-CoV-2 assay as a complementary tool for rapid presumptive identification of variant Alpha in the four laboratories using this assay as the primary diagnostic method for SARS-CoV-2 detection.

Study design
A total of 1139 routine samples positive in the Allplex SARS-CoV-2 assay (Seegene, Seoul, Korea) collected from 1 January to 31 March 2021, were included in the genomic surveillance protocol and analyzed in parallel using the rTEST COVID-19 qPCR B.1.1.7 assay (MultiplexDX, Bratislava, Slovakia) and WGS. For WGS extracted viral RNA from the positive samples was stored at -80 • C until transport and processed in batches once a week by the core laboratory. The study was approved by the institutional review board. The principles outlined in the Declaration of Helsinki were followed for the study and no additional samples were collected for the purpose of the study.

RT-PCR testing
The Allplex SARS-CoV-2 assay (Seegene, Seoul, Korea) was used as a routine diagnostic RT-qPCR in four clinical microbiology laboratories of the National Laboratory for Health, Environment and Food (NLZOH). In this assay, four sets of primers are used to detect the presence of the E, N, S and RdRP genes of SARS-CoV-2, with the latter two detected on the same fluorescent channel. The results are automatically interpreted by the supplied software Seegene Viewer (Seegene, Seoul, Korea), while the amplification curves can be additionally analyzed visually simultaneously or retrospectively. The assay was performed according to the manufacturer's instructions.
The rTEST COVID-19 qPCR kit was used to detect the B.1.1.7 variant. The assay aims to detects two of the NTD deletions, H69-V70 and Y144, characteristic of the B.1.1.7 variant. When these deletions are not present, amplification of the specific B.1.1.7 target versus the consensus target of the rTEST COVID-19 qPCR assay is delayed by at least five cycles, as a recommended cut-off by the manufacturer.
Amplification curve pattern analysis The shapes and relative positions of the amplification curves of the E and S/RdRP genes for each sample were subsequently checked in the software Seegene Viewer. The curves were visually analyzed by four different operators, one per laboratory, and classified into three different patterns according to the criteria shown in Figure 1. Briefly, pattern 1 consisted of the delay between the E and S/RdRP gene curve at the inflection point and lower end-point fluorescence in the S/RdRP gene curve compared with E gene curve, pattern 2 consisted of the sigmoidal E gene and linear S/RdRP gene curve amplification and higher end-point fluorescence in the S/RdRP gene curve and pattern 3 consisted of any other curve presentation. Pattern 1 and pattern 2 were considered predictive of variant Alpha. The results of the WGS were blinded to the operators and were revealed after the analysis.
Whole-genome sequencing Samples were obtained as RNA extracts and stored at -80 • C until further processing. Libraries were prepared with amplification enrichment using the ARTIC v3 kit according to protocol V.4 with recommended reagents and consumables. Finally, samples were sequenced on either the Illumina MiSeq or NovaSeq platform depending on availability. Consensus genome sequences were obtained by: mapping fastq reads to the reference genome (NC 045512) using BWA aligner (v 0.7.17); 2) implementing the ivar (v 1.3)/samtools (mpileup, v 0.1.19) workflow to filter the quality of aligned reads, remove primer sites and generate the consensus sequence. PANGO lines were assigned using the command-line tool PLEARN (v1.2.13), which allows a maximum of 10% Ns.

Statistical analyses
To evaluate the diagnostic performance of both the rTEST COVID-19 qPCR B.1.1.7 assay and the Allplex SARS-CoV-2 amplification curve pattern analysis in detecting variant Alpha, the sensitivity, specificity, positive predictive value (PPV) and negative predictive value for each method were calculated using WGS as a reference. A receiver operating characteristic curve was generated to verify the cut-off value set by the manufacturer of the rTEST qPCR COVID-19 kit. Analysis in GraphPad Prism (GraphPad Software, CA, USA) resulted in a table of possible cut-off values and their respective sensitivities and specificities with 95% CIs compared with WGS. The cut-off value with the highest Youden's index (sensitivity + specificity -1) was considered optimal. A p-value of 0.05 was considered statistically significant. Predictive values were determined using GraphPad QuickCalcs (GraphPad Software, CA, USA). Agreement of results between amplification curve pattern analysis and WGS was assessed using Cohen's kappa statistics with the following interpretive guidelines: no agreement (0.00), low agreement (0.00-0.20), fair  agreement (0.21-0.40), moderate agreement (0.41-0.60), substantial agreement (0.61-0.80) and near perfect agreement (0.81-1.00). The analysis was performed using GraphPad QuickCalcs.

Results
A total of 17 variants of SARS-CoV-2 were characterized among 1139 RT-PCR-positive samples ( The receiver operating characteristic curve to determine the optimal cut-off value of the rTEST qPCR assay is shown in the Supplementary Material. This value represents the maximum number cycles for which the amplification of the B.1.1.7 target on the S gene is delayed compared with the consensus target in a variant Alpha sample. The area under the curve was 0.9833 (95% CI: 0.9692-0.9975; p < 0.0001). The cut-off of 5.1 cycles was selected and the evaluation of the performance of rTEST qPCR assay was based on this cut-off value.
The sensitivity, specificity, PPV and negative predictive value using WGS as reference are shown in Table 2. Predictive values were calculated based on the prevalence of 13.6% relevant throughout the observation period and the prevalence of 49.5% calculated during the peak prevalence of variant Alpha in March 2021. The agreement of the results between the amplification curve pattern analysis and the WGS was satisfactory when the results of all participating NLZOH laboratories were considered (Table 3).

Discussion
For rapid determination of the specific SARS-CoV-2 variant, several screening protocols can be used based on the detection of nonspecific changes in diagnostic PCR target signals or on the detection of specific mutations associated with a particular viral variant. Failed positive spike gene signals have been evaluated and used in several countries facing rapid spread of variant Alpha (B.1.1.7) [12,13,15,16]. However, gene amplification failure may have other causes, and this approach is less specific than PCR assays designed to detect a specific mutation or set of mutations. Here, we show good analytical performance of both approaches for variant Alpha with way over 97% sensitivity and specificity.
Although the two most common lineages present in our sample B.1.258.17 and B.1.1.7 share one of the two NTD deletions that the rTEST q PCR COVID-19 B.1.1.7 kit is designed to detect, namely H69-V70, we have shown that the assay is still able to distinguish between these variants and detect B.1.1.7 with 98% specificity at the selected cut-off value. To our knowledge, no other independent evaluation of this assay has been published to date.
As additional variants of concern and variants currently under investigation have been introduced in Slovenia in the last 6 months, screening for only one of these variants was no longer a sufficient approach. The observation that the curve of the S/RdRp genes deviated from the expected shape of the curve, leading to possible prescreening of variant Alpha, was reported in the technical report of the manufacturer of the Allplex SARS-CoV-2 assay in February 2021 and later published by Italian authors [13]. In only four samples in this study, the presence of variant Alpha was confirmed by WGS. The authors therefore concluded that visual analysis of amplification curve profiles could be used mainly as a tool for retrospective studies.
In addition to the previously described delay in S/RdRP amplification, we found linearization of the S/RdRP gene amplification curve to be a separate and equally important change. Although, the interpretation criteria were descriptively defined, the combination of both patterns proved to be 97% sensitive and specific compared with WGS. Different analysts independently obtained satisfactory agreement of results with WGS in four different laboratories.
Finally, one of the advantages of systematic genomic monitoring performed using WGS is also the ability to compare other laboratory methods and procedures to the gold standard. High-quality evaluation of the analytical performance of diagnostic tests and procedures allows laboratories to adjust their surveillance and diagnostic protocols and respond in a timely manner to new epidemiological circumstances.
There are several limitations in our study. First, we were unable to determine the underlying molecular mechanism for the change in amplification curves; it is most likely due to a lower efficiency of the PCR reaction. Second, the specificity of such a nonspecific screening necessarily depends on the biological background of variants in a given geographic area, and cannot necessarily be extrapolated to other geographic locations. Third, following the rapid evolution of SARS-CoV-2 variants, the clinical utility of our results may be limited to areas where variant Alpha is still present, and to laboratories using a particular diagnostic assay that was also used in our situation. Fourth, there were some differences between sites in kappa. This can be explained by geographic variations of the virus, possible coinfection with different variants or by the subjective nature of the pattern analysis itself. Nevertheless, we report here the largest evaluation of its kind with more than 1000 SARS-CoV-2 genomes completely sequenced.

Conclusion
We describe here a good analytical performance of the two methods for the preliminary identification of SARS-CoV-2 variant Alpha. They can both be used as screening tests and even as confirmatory tests in areas with high prevalence of this lineage. Considering the still high prevalence of this variant in several regions and an increase in the prevalence of other variants of concern, similar approach can be used to monitor the prevalence of known SARS-CoV-2 variants in a cost-effective manner and save some WGS resources.

Future perspective
Genomic surveillance of the SARS-CoV-2 pandemic is important for planning, implementation and analysis of containment measures. Cheaper and faster approaches are needed for surveillance, especially in resource-limited settings. Simultaneous detection of the virus and typical variants is a promising and feasible approach, considering that the major circulating variants in a geographical area are known.

Summary points
• The accuracy of two PCR-based techniques for the detection of variant Alpha (B.1.1.7) in a large multicenter prospective cohort in Slovenia from January to March 2021 with 1137 positive samples were evaluated. • A specific mutation-based assay (rTEST COVID-19 qPCR B.1.1.7 assay) and nonspecific curve pattern analysis of the Allplex SARS-CoV-2 assay were compared with whole-genome sequencing. • Variant Alpha was detected in 155 samples (13.6%). Sensitivity and specificity were high for both assays, 98.1 and 98.0% for the specific assay and 97.4 and 97.5% for the nonspecific analysis, respectively. • The positive predictive value of both tests was 97.4 and 97.7%, respectively, with a prevalence of 49.5%.
• A good analytical performance of both methods was confirmed for the preliminary identification of SARS-CoV-2 variant Alpha and both can be used as screening and even as confirmatory tests in areas with high prevalence of this lineage. • This principle of cost-effective way to screen SARS-CoV-2 populations is also applicable to other emerging variants and may help to conserve some whole-genome sequencing resources.

Supplementary data
To view the supplementary data that accompany this paper please visit the journal website at: www.futuremedicine.com/doi/ suppl/10.2217/fmb-2021-0288 Author contributions Financial & competing interests disclosure